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A  collaboration  between  the  SEI  and  AMI  AIR — Team  Process  Integration  (TPISM) — is  currently  underway.  The  TPI 
effort  leverages  the  PSP  and  TSP  research  and  body  of  practice.  This  article  discusses  the  progress  and  performance  through 
a  pilot  project  with  the  MI  '-8B  Systems  Engineering  team  as  well  as  others  within  AMT  AIR  that  have  utilised  TPI  in 
non-software  domains.  This  article  will  share  lessons  and  experiences  with  other  industry  I  government  organisations  interest¬ 
ed  in  applying  the  TSP  in  a  non-software  setting.  The  early  results  suggest  some  encouraging  trends. 


Since  the  emergence  of  software  engi¬ 
neering  in  the  1 960s,  the  size,  pervasive¬ 
ness,  and  complexity  of  software-intensive 
systems  have  increased  by  several  orders  of 
magnitude.  The  size  of  aircraft  software 
systems  in  the  1960s  approximated  1,000 
lines  of  code  while  aircraft  systems  built  in 
2000  contained  more  than  six  million  lines 
of  code.  The  pervasiveness  of  software 
within  aircraft  systems  has  increased  from 
controlling  less  than  10  percent  of  die 
functions  die  pilot  performed  in  die  1960s 
to  80  percent  in  2000  (as  shown  in  Figure  1 
on  the  following  page). 

We  know  that  increases  in  software  and 
system  size  contribute  to  increased  com¬ 
plexity  which,  in  turn,  has  contributed  to 
pushing  delivery  and  costs  well  beyond  tar¬ 
geted  schedules  and  budgets  [1]. 

In  a  recent  workshop  conducted  by  die 
National  Defense  Industrial  Association, 
die  top  issues  relative  to  die  acquisition  and 
deployment  of  software-intensive  systems 
were  identified.  Among  them  are: 

•  The  impact  of  system  requirements 
upon  software  is  not  consistently  quan¬ 
tified  and  managed  in  development  or 
sustainment. 

•  Fundamental  systems  engineering  deci¬ 
sions  are  made  without  full  participa¬ 
tion  of  software  engineering. 

•  Software  life-cycle  planning  and  man¬ 
agement  by  acquirers  and  suppliers  is 
ineffective. 

So  the  biggest  challenge  is  creating  die 
right  foundation:  estimation,  planning, 
development,  and  management  practices  as 
well  as  team  processes,  training,  coaching, 
and  operational  support  that  will  assist  in  a 
migration  from  buggy  products  and  unnec¬ 
essary  rework  (resulting  in  inflating  devel¬ 
opment  costs)  to  a  proactive  approach  that 
builds  integrated,  quality  software-intensive 
systems  from  requirements  to  field  deploy¬ 
ment. 
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Background 

The  SEI’s  TSP  provides  engineers  with 
a  structured  framework  for  doing  soft¬ 
ware  engineering  work.  It  includes 
scripts,  forms,  measures,  standards,  and 
tools  that  show  software  engineers  how 
to  use  disciplined  processes  to  plan, 
measure,  and  manage  their  work  [2]. 
The  principal  motivator  for  the  TSP  is 
the  conviction  that  engineering  teams 
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can  do  extraordinary  work  if  they  are 
properly  formed,  suitably  trained, 
staffed  with  skilled  members,  and  effec¬ 
tively  coached  and  led. 

The  TSP  is  already  being  used  with 
great  results  on  software  teams  [3],  A 
Microsoft  study  reported  that  by  using 
the  TSP,  software  teams  cut  schedule 
error  from  10  to  one  percent.  With  its 
TSP  teams,  Intuit  has  increased  by  50 
percent  the  time  that  teams  can  spend 
in  developing  products  during  a  typical 
year-long  release  cycle:  Increased  quali¬ 
ty  has  dramatically  cut  the  testing  time 
required.  An  analysis  of  20  projects  in 


13  organizations  showed  TSP  teams 
averaged  0.06  defects  per  thousand 
lines  of  new  or  modified  code. 
Approximately  one-third  of  these  pro¬ 
jects  were  defect-free.  Other  studies 
show  that  TSP  teams  delivered  their 
products  an  average  of  just  six  percent 
later  than  planned.  This  compares 
favorably  with  industry  data  showing 
that  more  than  half  of  all  software  pro¬ 
jects  were  more  than  100  percent  late — 
or  were  cancelled.  These  TSP  teams 
also  improved  their  productivity  (size 
of  developed  code  per  hour  of  devel¬ 
opment  time)  by  an  average  of  78  per¬ 
cent. 

NAVAIR  develops,  acquires,  and 
supports  the  aircraft  and  related 
weapons  systems  used  by  the  U.S.  Navy 
and  Marine  Corps.  In  recent  years,  inter¬ 
est  in  applying  TSP  to  non-software 
domains  has  increased.  The  SEI  TSP 
team  has  collaborated  with  NAVAIR  to 
expand  the  TSP  to  teams  that  do  other 
engineering  along  with  software.  These 
include  areas  such  as  systems  engineer¬ 
ing  and  integration,  product  integrity, 
CM/DM/QA  (Configuration  Manage¬ 
ment/Data  Management/ Quality  Assur¬ 
ance),  and  process  improvement  itself. 

NAVAIR  already  has  a  proven  track 
record  with  the  TSP  and  has  demon¬ 
strated  return  on  investment  on  their 
software  projects  [4,  5],  Table  1  (on  the 
following  page)  shows  TSP  results  from 
two  NAVAIR  programs:  the  AV-8B’s 
Joint  Mission  Planning  System  (JMPS) 
program  and  the  P-3C  program.  This 
result,  due  to  the  reduction  in  defect 
density,  is  a  gross  savings  of  $3,225,606 
($3,782,153  less  the  investment  of 
$556,547).  In  turn,  the  ROI  is  derived 
from  the  cost  savings  compared  to  the 
cost  of  initially  putting  the  TSP  in 
place;  in  this  case,  it  was  a  ratio  of  bet¬ 
ter  than  7  to  1 .  Further,  these  organiza¬ 
tions  each  reached  CMM  Level  4  in  less 
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Figure  1 :  Increasing  Capabilities  and  Challenges  of  Software  in  DoD  Systems 1 


than  30  months — instead  of  the  typical 
six  years. 

Very  similar  results  occurred  with 
other  programs  at  that  time,  like  with  the 
E-2C  aircraft  program,  also  achieving 
CMM  Level  4  in  less  than  30  months 
with  their  development  teams  using  the 
TSP  at  the  same  time.  Most  recently  (Jan. 
2010),  the  HI  aircraft  program  worked 
less  than  20  months  to  obtain  a  CMMI 
Level  3  rating  while  their  development 
team  used  TSP  to  maintain  aircraft  soft¬ 
ware  for  the  fleet. 

The  organizations  referenced  have 
standardized  the  TSP  for  all  of  their 
software  development  and  maintenance 
work.  These  early  adopters  of  the  TSP 
are  meeting  their  mission  of  producing 
higher  quality  products  while  maintain¬ 
ing  significant  cost  savings.  Their  devel¬ 
opment  teams  now  like  using  the  TSP, 
saying  of  their  staffs,  “Once  they  have 
adopted  it,  they  can’t  imagine  working 
any  other  way.”  In  all  presented  cases,  the 
initial  investment  was  returned  in  their 
first  project  and  has  then  gone  forward 
time  and  again  to  benefit  the  organiza¬ 
tions  for  many  years. 

Results  from  these  examples  continue 
to  inspire  other  NAVAIR  System  Support 
Activities  (SSAs)  to  use  the  TSP.  There  are 
more  than  20  additional  NAVAIR  SSAs 
now  pursuing  software  process  improve¬ 
ment  activities.  NAVAIR  is  seeing  recurring 
savings  and  can  now  direct  cost  savings  to 


the  procurement  of  additional  aircraft  and 
weapons.  In  addition,  NAVAIR  used  the 
TSP  to  accelerate  CMMI  improvement. 

Starting  TPI  Efforts 

Based  on  the  demonstrated,  measured 
success  of  software  projects  using  the 
TSP  in  NAVAIR,  other  teams  asked  if 
they  could  apply  the  same  processes  to 
systems  engineering  and  software/ systems 
acquisition  projects.  As  a  result,  NAVAIR 
has  teamed  with  the  SEI  to  expand  the 
TSP  framework  to  a  technology  called 
TPI.  The  SEI  is  also  receiving  additional 
requests  to  apply  the  TSP  to  non-software 
settings  since  it  is  becoming  increasingly 
difficult  to  solve  software  problems  with¬ 
out  addressing  systems  engineering  issues. 

The  NAVAIR/ SEI  collaboration 
entails  testing  the  hypothesis  that  we  can 
achieve  the  same  kind  of  performance 
improvements  applying  TPI  to  systems 
engineering  as  we  did  applying  the  TSP  to 
software  projects,  thereby  improving  man¬ 
agement  and  communications  in  software¬ 
intensive  systems  and  acquisitions.  Our 
approach  will  entail  conducting  a  series  of 
pilot  projects  to  determine  if  extending 
TSP  practices  to  systems  engineering 
results  in  measurable  improvement.  We 
will  then  use  the  results  of  this  work  to 
establish  common  processes  for  both  sys¬ 
tems  and  software  engineering  across  the 
NAVAIR  teams.  Initially,  the  AV-8B  Joint 
SSAs  (developing  the  Harrier  Aircraft) 


was  selected  as  the  systems  engineering 
pilot  program. 

In  kicking  off  these  efforts,  we  real¬ 
ized  that  there  were  a  number  of  research 
challenges  that  specifically  had  to  be 
addressed.  We  extended  the  TSP  practices 
to  systems  engineering  by: 

•  Determining  the  baseline  performance 
for  systems  engineering  work  at 
NAVAIR. 

•  Developing  prototype  processes/ 
process  definitions/ scripts  for  systems 
engineering. 

•  Formulating  relevant  measures,  espe¬ 
cially  size  and  quality  measures  perti¬ 
nent  to  systems  engineering. 

•  Building  conviction  and  discipline  in 
our  leadership  and  team  member  train¬ 
ing  materials  for  teams  that  don’t  nec¬ 
essarily  write  software  programs. 

•  Developing  an  extensible  tool  that 
allows  for  outlining  any  process,  for 
collecting  data  unobtrusively,  and  for 
defining  a  measurement  framework 
pertinent  to  any  engineering  domain. 
Early  results  of  applying  TPI  show 

some  encouraging  trends.  The  AV-8B 
Systems  Engineering  pilot  project  team  is 
changing  the  way  they  do  their  work  and  is 
beginning  to  see  some  results  similar  to 
those  realized  by  TSP  teams.  The  AV-8B 
team  is  practicing  more  disciplined  meth¬ 
ods  for  planning  and  executing  their  work. 
They  are  meeting  their  missions  and 
beginning  to  see  some  cost  savings.  In 
addition,  the  pilot  team  is  inspiring  other 
NAVAIR  4.0  SSAs  to  pursue  process 
improvement  [6], 

Benefits 

Through  the  pilot  effort,  we  are  seeing 
some  of  the  following  benefits: 

Establishment  of  a  Systems 
Engineering  Baseline 

We  are  beginning  to  establish  a  baseline 
for  systems  engineering  performance  at 
NAVAIR  that  can  be  used  for  estimating, 
planning,  and  tracking  projects  and  pro¬ 
grams: 

•  The  requirements  productivity  rate 
varies  between  three  and  nine  require  - 


Table  1:  TSP  Results  at  NAS  AIR 


Project 

Defect  Density 
Before  TSP 

Defect  Density 
After  TSP 

Total  Defects 
Before  TSP 

Total  Defects 
After  TSP 

Average 
Cost  to  Fix 

Product  Size 
(KSLOC) 

Cost  Savings 
From  Reduced 
Defects 

AV-JMPS 

1.13 

0.59 

501 

261 

$8,330 

443.0 

$1,992,663 

P-3C 

4.60 

0.60 

176 

23 

$8,432 

38.3 

$1,789,490 

Total  Savings:  $3,782,153 
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ment  statements  per  hour,  depending 
on  the  complexity  of  the  project2. 

•  By  just  tracking  requirements  size 
growth,  tire  team  was  able  to  decrease 
die  rate  of  project  size  growdi  from 
23.6  percent  in  die  initial  development 
cycle  to  1 1 .5  percent  in  the  subsequent 
development  cycle. 

•  By  collecting  the  planned  and  actual 
requirements  size  and  growth  for  the 
various  components  and  the  team  pro¬ 
ductivity  rate,  the  team  builds  up  his¬ 
torical  data  that  can  be  used  on  future 
projects. 

•  To  quote  one  team  leader:  “Prior  to 
TPI,  we  made  estimates  in  a  bubble. 
Now  we  are  establishing  and  maintain¬ 
ing  baselines  for  all  of  our  releases, 
which  allow  us  to  make  better  esti¬ 
mates  and  more  realistic  plans  and 
schedules.” 

Establishment  of  Planning  Practices 

Planning  at  die  program  and  team  level  is 
now  accomplished  by  holding  multi-team 
launches  that  involve  all  of  the  teams 
implementing  either  the  TSP  or  TPI.  At 
first,  tliey  plan  for  no  more  than  four 
months  of  work  at  a  time  so  that  dieir 
tasks  can  be  detailed  enough  with  fairly 
stable  component  sets.  The  component 
sets  start  to  vary  for  a  longer  development 
duration  so  their  plan  would  be  less  stable. 
This  process  is  used  by  the  AV-8B  pro¬ 
gram  to  understand  requirements  from 
management,  assemble  plans,  allocate 
work,  and  achieve  commitment  to  plans 
from  management  and  team  members. 
The  overall  plan  for  the  year  and  the  next- 
phase  plan  are  developed  by  the  teams, 
work  is  allocated  by  the  team,  and  the 
schedule  is  determined  and  committed  to 
by  team  members. 

Establishing  Tracking  Practices 

For  tracking  purposes,  work  is  broken 
down  into  small  chunks  that  can  easily  be 
tracked  (tasks  are  tracked  at  a  granularity 
of  less  than  10  hours).  Tracking  only  the 
task  hours  per  week  (planning  for  around 
20)  allows  two  or  three  tasks  to  be  com¬ 
pleted  each  week.  Work  is  tracked  daily  by 
team  members  and  discussed  weekly  in 
team  meetings:  Every  team  member 
knows  how  tliey  are  performing  to  their 
individual  plan  and  the  team  plan. 
Monthly  status  reports  are  derived  from 
die  consolidated  weekly  reports  by  the 
team  leader  and  presented  to  the  integrat¬ 
ed  product  team  leads. 

Twelve  team  members  were  able  to 
achieve  (on  average)  between  18  and  22 
on-project  task  hours  per  week.  The  team 
performed  well  above  the  planned  task 


hours:  15  per  week  in  the  first  cycle. 

The  engineers  embraced  project  plan¬ 
ning  and  tracking.  Each  individual  is  able 
to  track  personal  commitments  to  the 
team,  enabling  the  team  to  better  monitor 
commitments  to  the  program.  Tracking 
the  work  helped  the  team  members  with 
staying  on-task,  commenting  that:  “I  need 
to  stop  doing  X  to  get  back  on  track.  It  is 
very  easy  to  see  the  impact  daily  and  week¬ 
ly  of  not  working  to  the  plan.” 

Developing  Standard  Processes, 
Measures,  and  Tools 

Standard  processes,  measures,  terminol¬ 
ogy,  and  tools  were  developed  and  used 
by  the  AV-8B  Program: 

•  The  PSP-derived  Excel  spreadsheet 
and  a  process  support  technology 
Access-based  tool  were  used  for  esti¬ 
mating,  planning,  and  tracking  work 
for  team  members  and  team  leads. 

•  Team  members  identified,  defined, 
and  documented  all  systems  engi¬ 
neering  standard  life-cycle  processes 
in  the  tool.  The  team  defined  and 
developed  an  18-step  overall  systems 
engineering  process  and  a  482-step 
detailed  systems  engineering  process. 

•  Through  the  defined  processes, 
NAVAIR  was  able  to  maintain  the 
consistency  of  processes  across  pro¬ 
jects/programs.  The  defined  processes 
also  offered  the  ability  to  cross-train 
individuals.  One  integrated  product 
team  lead  said:  “We  have  a  team  con¬ 
cept  across  our  program  with  all  of  the 
sub-teams  (systems  engineering,  prod¬ 
uct  integrity,  software,  test,  lab,  etc.). 
We  also  have  a  common  set  of 
processes  and  metrics  to  help  all  of  the 
teams  better  communicate  and  address 
dependencies  across  the  teams.” 

Performance  Trends 

With  no  historical  data  to  go  by,  the  team’s 
initial  plan  identified  a  guess  at  a  goal  of 
less  than  5  percent  schedule  slip  and  mea¬ 
sured  performance  against  die  goal.  The 
actual  performance  had  an  overrun  of  less 
than  10  percent.  Now  with  some  historical 
data,  the  team  can  set  more  realistic  goals 
and  try  to  continually  improve  on  them. 
As  far  as  cost  and  quality  performance, 
size  and  effort  estimates  were  within  +10 
percent  of  what  was  planned,  and  there 
were  no  high-priority  problem  reports 
coming  out  of  test. 

Employee  Work/Life  Balance 

TPI  helped  improve  employee  work/life 
balance.  In  order  to  get  their  job  done 
before  implementing  TPI,  employees  rou¬ 
tinely  worked  overtime.  With  TPI  (and  in 
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order  to  get  their  18-22  task  hours  per 
week),  they  did  not  have  to  work  as  much 
overtime.  Overtime  was  decreased  from 
being  standard  practice — sometimes  25 
percent  or  more — to  occasional  overtime 
hours  (less  than  10  percent). 

Customer  Responsiveness 

Customer  responsiveness  has  improved  to 
tire  fleet,  tire  naval  aviators,  and  to  the 
internal  program  managers.  The  systems 
engineering  team  is  able  to  more  easily 
adapt  to  program  and  personnel  changes. 
The  pilots  are  beginning  to  provide  input 
early  on  in  the  project — during  die  launch 
process — before  the  work  has  com¬ 
menced  (instead  of  providing  feedback 
during  the  test  phases).  Program  manage¬ 
ment  feels  drat  the  TSP/TPI  efforts  are  a 
success  because  the  teams  understand 
their  work  and  the  dependencies  among 
all  of  the  teams.  The  systems  engineering 
team  can  also  plan  for  a  percentage  of 
unplanned  tasks  to  use  their  data  to  nego¬ 
tiate  impact  and  trade-offs  of  unplanned 
work  to  planned  work. 

More  Teams  Doing  TPI 

We  have  since  launched  more  non-soft¬ 
ware  teams  using  the  TPI  approach.  One 


of  these  is  a  mixed  engineering  team  at 
Joint  Munitions  Effectiveness  Matrix 
Weaponeering  Systems  that  is  applying 
the  TPI  to  their  non-software  work  as 
well  as  to  their  software  team.  This  team 
has  been  using  TPI  for  more  than  a  year, 
has  gone  through  four  launch/relaunch- 
es,  and  has  seen  the  types  of  benefits 
that  the  AV-8B  team  has  seen.  They  also 
are  seeing  steady  progress  in  making 
more  accurate  and  precise  estimates  of 
their  work,  and  have  refined  the  triggers 
that  would  initiate  an  adjustment  of  their 
behavior  so  they  stay  on  schedule. 

Another  example  is  the  Precision 
Attack  Weapon  System  Tactical  Program 
Office,  demonstrating  the  effectiveness 
of  the  TPI  approach  for  one  of  their 
systems  engineering  teams.  Their  team 
has  been  using  the  TPI  approach  for 
more  than  a  year  and  saw  immediate 
benefits.  During  the  initial  launch,  they 
developed  a  never-before-seen  detailed 
plan  that  gave  senior  management  the 
needed  data  to  get  additional  project 
funding  without  having  to  arm  wrestle  the 
Program  Manager,  Air  (PMA). 

Then  there  is  the  P-3  lab  team  at  the 
Patuxent  River,  Maryland  Naval  Air 
Station,  who  has  been  applying  this 


approach  to  the  many  configurations  of 
the  lab  setup  they  must  provide.  The  P-3 
team  started  applying  TPI  as  an 
approach  to  the  implementation  phase  of 
their  Black  Belt  DMAIC  (or  Define, 
Measure,  Analyze,  Improve,  and 
Control)  project.  Since  starting  about 
three  years  ago,  the  team  has  since  pro¬ 
vided  two  annual  cycles  of  lab  services 
and  is  halfway  through  their  third.  The 
P-3  lab  team  supports  their  customers  by 
providing  more  than  a  dozen  lab  config¬ 
urations  across  the  PMA.  This  breaks 
into  two  basic  types  of  support:  usage  in 
terms  of  running  tests,  and  support  in 
terms  of  configuring  labs  for  those  tests. 
Aggregate  lab  usage  data  shows  a  devia¬ 
tion  of  12  percent  less  than  planned 
while  aggregate  lab  support  data  shows  a 
deviation  of  .5  percent  more  than 
planned.  While  performance  is  impres¬ 
sive,  deviation  was  at  times  greater  when 
examined  at  the  individual  lab-customer 
level.  As  expected,  this  aggregate  devia¬ 
tion  demonstrates  the  advantage  of  esti¬ 
mating  in  smaller  increments. 

At  the  time  of  writing  this  article, 
several  other  process  improvement 
efforts  at  NAVAIR  are  getting  started 
with  plans  of  applying  TSP  to  their  soft¬ 
ware  teams  and  TPI  to  their  non-soft¬ 
ware  teams. 

Summary 

All  engineering  efforts  must  start  with 
integrated  teams.  These  teams  must  plan 
their  work — and  work  to  those  plans — 
while  collecting  basic  measures.  They 
must  then  apply  analyses  to  this  data  and 
derive  metrics  to  determine  their  status  on 
current  work  and,  eventually,  as  a  source 
for  improving  their  planning  capability  on 
future  work.  From  this  approach,  we  have 
seen  quality  products  and  services  deliv¬ 
ered  over  and  over  with  the  potential  for 
further  improvement. 

To  make  this  happen,  we  have  seen  the 
need  to  put  in  place  the  TPI  foundation  of 
estimation  and  planning  processes,  team 
processes,  development  and  management 
practices,  effective  and  timely  training,  as 
well  as  launch,  coaching,  and  operational 
support. 

Projects  that  have  adopted  these  meth¬ 
ods  have  shown  a  dramatic  increase  in 
product  quality  and  fidelity  of  schedule 
and  effort  estimates.  The  methods  are 
supported  by  a  doctrine  that  trains  and 
sustains  performance  and  quality  im¬ 
provement  in  an  organization. 

This  article  has  shown  what  is  possi¬ 
ble  when  teams  use  TPI  to  establish  this 
foundation  to  meet  critical  business 
needs.  The  end  result  is  the  delivery  of 
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Software  Defense  Application 

Software  defense  organizations  will  benefit  by  learning  about  Team  Process 
Integration,  die  continuing  collaboration  between  the  SEI  and  NAVAIR.  As  detailed 
in  the  article,  results  from  current  projects  utilizing  TPI  show  a  gross  savings  of  more 
than  $3.7  million  and  a  net  savings  of  more  than  $3.2  million,  with  a  return  seven 
times  the  original  investment.  Quality  improvement  on  two  examined  projects  was  a 
reduction  in  defect  density  from  1.1  to  0.59  defects  per  diousand  LOC  on  one  and 
4.6  to  0.6  defects  per  diousand  LOC  on  the  other.  TPI  lowers  costs,  helps  projects 
meet  schedules,  and  improves  productivity. 


high  quality  systems,  on  cost,  and  with 

improved  productivity.^ 
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Notes 

1.  This  graphic  was  created  based  on  a 
table  called  “System  Functionality 


Requiring  Software,”  but  die  original 
creator  of  die  table  is  debated:  either 
PM  Magazine  or  a  US.  Air  Force  “Bold 
Strike”  Executive  Software  Course 
from  1992.  To  view  the  table,  see: 
Ferguson,  Jack.  “Crouching  Dragon, 
Hidden  Software:  Software  in  DoD 
Weapon  Systems.”  IEEE  Software 
July/ Aug.  (2001):  105-107. 

2.  For  example,  AV-8B  uses  Telelogic 
DOORS  Objects  to  identify  die  num¬ 
ber  of  requirement  statements  and, 
hence,  the  size  of  the  requirement  set. 
Any  organization/program  product 
can  be  viewed  as  a  comparable  proxy. 
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